System and method for creating photo books using video

ABSTRACT

A system and method are described for creating photo stories using video. The method includes receiving a user request to upload a user-specified video stream; determining whether the video stream meets one or more predetermined requirements, uploading the video stream; analyzing the video stream to select video frames for extraction based on a set of predetermined frame selection criteria; extracting the selected video frames from the video image; selecting a story book template to be used with the video images; and generating a preview of the story book template with the video images within a graphical user interface, the user editing the story book template with text and/or moving or modifying the video frames within the preview.

CLAIM TO PRIORITY

This application is a continuation application of and claims priority tocommonly assigned co-pending U.S. patent application Ser. No.12/961,993, titled “System And Method For Creating Photo Books UsingVideo”, filed Dec. 7, 2010, which in turn is a continuation-in-part ofcommonly assigned U.S. patent application Ser. No. 12/942,478, filedNov. 9, 2010, entitled, “System And Method For Creating Photo StoryBooks”. The disclosures of the above applications are incorporatedherein by reference.

BACKGROUND

1. Field of the Invention

This invention relates generally to the field of network data processingsystems. More particularly, the invention relates to an improvedarchitecture and method for creating photo books using video.

2. Description of the Related Art

Current Web-based photo sharing systems allow end users to upload, shareand print digital photographs over the Internet. These systems alsoallow the end user to combine groups of related photos into printable“Photo Books” with various cover options, designs and templates. Usersmay select among different Photo Book design “themes” including, forexample, “Wedding” “New Baby,” and “Holidays.” Within each Photo Booktheme, the user may further select among different “style” templatesincluding different fonts, photo edges, page layouts, and coloredbackgrounds.

One limitation of current Photo Book systems, however, is that the usermust manually select an appropriate Photo Book theme template and thenmanually select the photos and styles to be used for the selectedtemplate. This results in a significant amount of time wasted selectingoptions which, depending on the requirements of the end user, mayultimately be inappropriate. By way of example, the end user maymanually choose a template which is suitable for displaying 1-5 photoswhereas the user may ultimately need a template capable of displaying,for example, 10 or more photos. Similarly, the user may wish to usephotos with different sizes and orientations, thereby rendering certaintemplates unusable. Additionally, current systems do not automaticallyselect templates based on various types of metadata associated with theuploaded photos including, for example, the dates on which the uploadedphotos were captured.

Moreover, current systems are incapable of generating photo books orother printable items (e.g., greeting cards, keepsakes, etc) using videocontent uploaded by a user. Given that most users now capture video aswell as still images it would be beneficial to provide options forgenerating photo books and/or other printable items using uploaded videocontent. Consequently, what is needed is an improved system and methodfor creating photo books and other printable items using video content.

SUMMARY

A system and method are described for creating photo stories usingvideo. For example, a computer-implemented method according to oneembodiment comprises: receiving a user request to upload auser-specified video stream; determining whether the video stream meetsone or more predetermined requirements, wherein if the video stream doesnot meet the one or more predetermined requirements, then denying therequest, and wherein if the else: uploading the video stream; analyzingthe video stream to select video frames for extraction based on a set ofpredetermined frame selection criteria; extracting the selected videoframes from the video image; selecting a story book template to be usedwith the video images; generating a preview of the story book templatewith the video images within a graphical user interface (GUI), the userediting the story book template with text and/or moving or modifying thevideo frames within the preview; and in response to detecting that theuser has completed editing the story book, saving the story book andgenerating a print job for printing the story book.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained from thefollowing detailed description in conjunction with the followingdrawings, in which:

FIG. 1 illustrates a system architecture for an online photo servicewhich includes a contacts database.

FIG. 2 illustrates a method according to one embodiment of theinvention.

FIG. 3 illustrates a system architecture for an online photo servicewhich includes a contacts database and a calendar database.

FIG. 4 illustrates one embodiment of the invention which automaticallyselects a photo story template and layout options for the photo storytemplate.

FIGS. 5 a-b illustrate a method for automatically selecting a photostory template.

FIG. 6 illustrates a method for laying out photos and text regionswithin a photo story template and for selecting colors for designelements.

FIGS. 7 a-9 illustrate embodiments of graphical user interface (GUI) forcreating and managing photo stories.

FIG. 10 illustrates a GUI for entering profile data for a user.

FIG. 11 illustrates a GUI for browsing for people within a photo storiesdatabase.

FIG. 12 illustrates a GUI for entering details for people within thephoto stories database.

FIG. 13 illustrates one embodiment of a GUI for creating and/or editinga photo story template.

FIG. 14 illustrates one embodiment of a GUI for creating and/or editinga photo story overlay.

FIG. 15 illustrates one embodiment of a GUI for creating and/or editinga photo story photo area.

FIG. 16 illustrates one embodiment of a GUI for creating and/or editinga photo story text area.

FIG. 17 illustrates one embodiment of a system architecture including aphoto story theme engine.

FIG. 18 illustrates a method according to one embodiment of theinvention.

FIG. 19-28 illustrate embodiments of a graphical user interface forgenerating a photo story book using a theme.

FIG. 29 illustrates one embodiment of a system for creating photo booksusing video.

FIG. 30 illustrates one embodiment of a method for creating photo booksusing video.

FIG. 31 illustrates a graphical user interface (GUI) employed in oneembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Described below is a system and method for creating and sharing videophoto stories over the Internet and using printed photo story books.Throughout the description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the present invention may be practicedwithout some of these specific details. In other instances, well-knownstructures and devices are shown in block diagram form to avoidobscuring the underlying principles of the present invention.

The assignee of the present application has developed an onlinestationery and greeting card system as described in the followingco-pending patent applications, which are incorporated herein byreference:

SYSTEM AND METHOD FOR CREATING PHOTO STORY BOOKS, Ser. No. 12/942,478,filed Nov. 9, 2010;

SYSTEM AND METHOD FOR MANAGING CONTACTS AND CALENDARS WITHIN AN ONLINECARD SYSTEM, Ser. No. 12/702,932, filed Feb. 9, 2010;

SYSTEM, METHOD AND GRAPHICAL USER INTERFACE FOR MANAGING CONTACTS ANDCALENDARS WITHIN AN ONLINE CARD SYSTEM, Ser. No. 12/703,051, Feb. 9,2010; and

SYSTEM, METHOD AND GRAPHICAL USER INTERFACE FOR MANAGING CONTACTS ANDCALENDARS WITHIN AN ONLINE CARD SYSTEM, Ser. No. 12/703,130, Feb. 9,2010.

Certain aspects of the systems described in these applications may beused for implementing an online photo service for creating, sharing andprinting groups of photos, as described in detail below. As such, thesystem architecture described in the co-pending applications will firstbe described, following by a detailed description of a system and methodfor creating and sharing photo stories.

EMBODIMENTS DESCRIBED IN CO-PENDING APPLICATIONS

FIG. 1 illustrates one embodiment of a system architecture importing andmanaging contacts within an online stationery service 200 and FIG. 2illustrates a corresponding method. One embodiment of the onlinestationery service 100 merges contact data from multiple differentsources and then converts the contact data into a format which isoptimized for online stationery mailing functions. A brief overview ofthe method illustrated in FIG. 2 will now be provided within the contextof the architecture shown in FIG. 1. It should be noted, however, thatthe underlying principles of the invention are not limited to thespecific architecture shown in FIG. 1.

At 201, a contacts import module 109 manages the importation of contactsfrom various local and/or online contact databases identified by the enduser. In the illustrated embodiment, the contacts import module 109comprises a format conversion module 104 and a conflict detection andresolution module 105. As shown in FIG. 1, the format conversion module104 reads contacts data from online contacts databases 101-102; localcontacts databases 103 (i.e., “local” to the user's client computer140); and/or existing contacts 111 already stored on the onlinestationery service 100 (e.g., the end user may have already establishedan account on the online stationery service 100 to send stationery andmay have entered information for a set of contacts 111). If theonline/local contact formats are supported, determined at 202, then at203, the format conversion module converts the contacts to a formatoptimized for use on an online stationery service 100. To perform theformat conversion, the format conversion module 104 parses the contactdata in source data structure (e.g., the CSV file, vCard file, etc.),extracts the data, and assigns the data to appropriate data fields inthe new data structure. Various well known techniques for convertingdata from one format to another may be employed by the format conversionmodule 104. Once converted (and following conflict detection describedbelow), the contacts data is stored in its new format within a contactsdatabase 110 on the stationery service. Various features associated withthis new data format are described in detail below.

At 204, a conflict detection and resolution module 105 merges the localand/or online contacts with existing contacts 111 already stored on theonline stationery service 100 and detects any conflicts which may resultfrom the merge operation. A conflict may result if one or more contactsbeing imported are already stored within the existing contacts database111. In such a case, the conflict detection and resolution module 105resolves the conflicts at 205 using a set of conflict resolution rules(described below). Once all conflicts have been resolved, the data ispersisted within the contacts database 110 and made accessible to endusers via the stationery service contacts manager 112. In oneembodiment, the contacts database 110 is implemented using mySQL.However, various different database formats may be employed while stillcomplying with the underlying principles of the invention (e.g.,Microsoft SQL, IBM SQL, etc).

At 207, the user identifies one or more “households” within thestationery service contacts database 110. As described below, householdsare specialized groups of contacts who live at the same address. Theconcept of a “household” is a particularly useful abstraction for anonline stationery service 100 which mails stationery on behalf of auser.

As illustrated in FIG. 1, in one embodiment, all operations to thestationery service contacts database 110 occur through the stationeryservice contacts manager 112. That is, the stationery service contactsdatabase 110 is used for persistent storage of contacts data containingthe features described herein and the stationery service contactsmanager 112 is the application-layer program code used to performoperations on the stationery service contacts database 110 as describedbelow. The presentation and session management logic 106 comprises theprogram code for maintaining user sessions and for dynamicallygenerating Web pages containing (among other things) the graphical userinterface (GUI) features for manipulating contacts data as illustratedherein.

Returning to the method of FIG. 2, at 207, the user selects andpersonalizes a stationery design. In one embodiment, this isaccomplished with a stationery personalization engine 120 such as thatdescribed in co-pending application entitled SYSTEM AND METHOD FORDESIGNING AND GENERATING ONLINE STATIONERY, Ser. No. 12/188,721, filedAug. 8, 2008, which is assigned to the assignee of the presentapplication and which is incorporated herein by reference. In oneembodiment, the stationery personalization engine 120 performs all ofthe functions described in the co-pending application as well as theadditional functions described herein (e.g., selectingcontacts/households for a stationery mailing via the stationery servicecontacts manager 112, selecting between a default message or a personalmessage for the contacts/households, etc).

At 208, the end user creates a default message to be used for astationery mailing and, at 209, the contacts and/or households for themailing are identified by the end user. If the user wishes to include apersonalized message in lieu of the default message for one or morecontacts/households, determined at 210, then the user selects acontact/household at 211 and enters the personalized message for thecontact/household at 212. If any additional personalized messages are tobe included, determined at 213, then steps 211 and 212 are repeateduntil all personalized messages have been entered.

At 214, all of the information related to the stationery order,including the selected stationery design, default messages, personalizedmessages and associated contacts and households are formatted forprinting by a print module 150 which generates a print job 155. Theformatting may include converting the stationery data mentioned aboveinto a format usable by a particular printer. By way of example, aletter press printer may require different formatting than a digitalpress printer. In one embodiment, the specifications for the print jobare encapsulated as metadata in an Extensible Markup Language (“XML”)document and transmitted to an external print service 152. In oneembodiment, the XML document includes a hyperlink (e.g., a URL) to theformatted print job 155 on the online stationery service 100. The printservice 152 then accesses the print job by selecting the hyperlink.Regardless of how the print job is accessed, at 215, the formatted printjob 155 is transmitted to either an internal printer 151 or an externalprint service 152 (e.g., over the Internet). Once printing is complete,the online stationery service 100 or the print service 152 mails thestationery to the contacts and/or households identified by the end user.

Having provided an overview of the method set forth in FIG. 2 and thearchitecture illustrated in FIG. 1, various specific details associatedwith managing contacts, generating print jobs and mailing stationeryfrom an online stationery service 100 will now be provided. It should benoted, however, that the underlying principles of the invention are notlimited to the particular architecture shown in FIG. 1 or the particularmethod set forth in FIG. 2.

FIG. 3 illustrates one embodiment of a system architecture whichintegrates contacts and calendar data and includes additional modulesfor generating reminders, filtered recommendations, and for schedulingdelivery of greeting cards/stationery. Specifically, in addition to thesystem components illustrated in FIG. 2, this embodiment includes acalendar service 301, a reminder service 302, a recommendation enginewith filtering logic 303 and a scheduling service 304. Thestationery/card service illustrated in FIG. 3 also includes a stationeryservice calendar database 310 for storing calendar data, a scheduledorders database 305 for storing order schedule data, a user database 310for storing user data (e.g., user stationery/card preferences,configuration options, etc.), and an accounts database 350 for storinguser account data. In one embodiment, the various databases shown inFIG. 3 are not actually separate databases but, rather, separate datastructures (e.g., tables) within a relational database.

In one embodiment, the calendar database 310 stores calendar data foreach user of the online stationery/greeting card service 200 and thecalendar service 301 comprises executable program code for managing thecalendar data (e.g., reading, adding, deleting, and modifying calendarentries). In one embodiment, the calendar service 301 also acts as aninterface to the calendar data to other system modules 212, 302, 303,and 304 (e.g., by exposing a calendar data API).

The reminder service 302 generates graphical or audible reminders ofupcoming calendar events and may prioritize the events based on a set ofprioritization rules. In one embodiment, the calendar events areprioritized chronologically but some events are given relatively higherpriority than other events based on the relationship between the userand the card/stationery recipients (e.g., the user's parents may begiven a higher priority than the user's friends, notwithstanding theevent dates). For example, an entry corresponding to Mother's Day may beprioritized at the top of the list even though other events (e.g., LaborDay) are nearer in time. In one embodiment, the highest prioritizedevent is either the next event created by the user (birthday,anniversary, other, etc) OR the next significant Holiday where“significant” holidays are identified in the online stationery/cardsystem and may change over time. In one embodiment, the “significant”holidays are Mother's Day, Father's Day, and Christmas.

The recommendation engine with filtering logic 303 generatesstationery/card recommendations to the end user based on the user'spreferences and allows the user to filter the results according touser-specified filtering criteria. In one embodiment, therecommendations are categorized based on certain stationery/cardcharacteristics and visually displayed to the end user in differentcategories (e.g., “new designs,” “with pictures,” etc.). Moreover, inone embodiment, the recommendation engine 303 recommends stationerydesigns based on the preferences of the user and/or the preferences ofthe recipient (if known).

In one embodiment, the scheduling service 304 implements a schedulingalgorithm to ensure that stationery/card orders are delivered within aspecified delivery window and/or on a specific date. For example, theuser may specify that a stationery/card order is to arrive 3-4 daysprior to a recipient's birthday. In such a case, the user does not wantthe card to arrive to soon (e.g., 2 weeks prior to the birthday) or toolate (after the birthday). To precisely schedule stationery/card orders,one embodiment of the scheduling service 304 evaluates the time requiredby the print services required to fulfill the order (e.g., thermography,digital press, etc.), the delivery type (e.g., regular mail, FedEx,etc.), and the end user preferences.

In one embodiment, three data points are used to determine the deliverydate: processing time, fulfillment time, and shipping transit time. Theprocessing time may be based on the type of order. For example,processing time can be 0 days for greeting cards and several days forsome stationery cards (e.g., those which require additional review bythe online card/stationery service prior to fulfillment). The processingtime is based on business days so it must factor in non-business dayssuch as Holidays and Weekends to determine the number of calendar daysrequired for processing. Fulfillment time is the number of days requiredto print, finish and ship/mail the order and is typically between 1-3days (e.g., depending on the printing requirements). This time is basedon business days for the fulfillment site which, in one embodiment, maybe different than business days for the processing site. Shippingtransit time is estimated based on the fulfillment site physicallocation and the shipping address of the recipient. The shipping transittime is based on business days for the shipping carrier and may bedifferent than business days for the processing site and fulfillmentsite. In one embodiment, after computing the sum of the three datapoints, the system has the number of calendar days required for theorder and determines the date that the order must be sent to theprocessing site in order to be delivered on the specified delivery date.

Presentation and session management logic 206 generates the Web-basedgraphical user interface (GUI) features described below, allowing theend user to view and edit the calendar data, contacts data, filteredcard recommendations, and scheduling data. As illustrated in FIG. 3, thepresentation and session management logic 206 communicates with each ofthe other functional modules and/or communicates directly with thestationery service databases 215 to retrieve the data needed for displaywithin the GUI. Embodiments of the Web-based GUI features generated bythe presentation and session management logic 206 are set forth below.

In one embodiment, each of the functional modules illustrated in FIG. 3exposes an application programming interface (API) to provide access todata managed by that module. For example, the contacts manager 212exposes an API allowing the calendar service 301 (and other modules) toaccess contacts data and vice versa. Alternatively, each of thefunctional modules may access the database(s) 215 directly.

In one embodiment, the calendar service 301 automatically generatescalendar events based on the contacts data stored within the contactsdatabase 210. By way of example, the calendar events may includebirthdays, anniversaries, and other significant milestones associatedwith each of the contacts in the contacts database 210. In addition, thecontacts manager 212 stores relationship data identifying therelationship between the user and each of the contacts in the user'scontacts database 210 (e.g., identifying the user's spouse, siblings,parents, children, etc.). The calendar service 301 uses the relationshipdata to generate calendar events. For example, if the relationship dataidentifies the user's mother and father, then the calendar data mayassociate Mother's Day and Father's Day, respectively, with thosecontacts. Similarly, if the user is married with children the calendarservice may associate his/her spouse with Mother's Day or Father's Dayand/or the user's wedding anniversary.

Once calendar events are scheduled, in one embodiment, the reminderservice 302 automatically generates reminders for upcoming events. Forexample, if a friend's birthday is approaching, then the reminderservice 302 will notify the user a specified number of days/weeks aheadof time, so that the user has time to send a card. The specific timingof the reminder notifications may be specified by the end user andstored along with other user preferences within the user database 311.

In one embodiment, the reminders are generated and displayed within aWeb-based GUI when the user logs in to the online stationery/cardservice 200 and/or may be sent to the user in the form of an emailmessage or mobile text message. If sent in an email, links to the onlinestationery/card service website may be embedded within the message toencourage the user to design a new card.

In one embodiment, the recommendation engine 303 generates greetingcard/stationery recommendations based on the occasion, the identity ofthe contact associated with the occasion, and the end user'spreferences. For example, if a particular contact's birthday isapproaching, the recommendation engine 303 may recommend certaingreeting card styles (e.g., modern, classical, etc.) based on thecontact's preferences and/or the user's preferences. The filtering logicallows the recommendations to be filtered based on specified variables(e.g., theme, color, card format, card size, number of photos, etc.).

Embodiments of a System and Method for Creating and Sharing PhotoStories

FIG. 4 illustrates one embodiment of a system for creating and sharingphoto stories. Some of the modules described above with respect to FIGS.1 and 3 are not illustrated in FIG. 4 to avoid obscuring the underlyingprinciples of the invention. However, the online stationery/card/photoservice 100 (hereinafter simply “photo service 100”) may implement allof these features concurrently on a single, integrated platform.

The illustrated embodiment includes a photo story template engine 401for separating uploaded photos into a plurality of “photo stories” basedon user input and metadata associated with each of the photos, andautomatically selecting an appropriate photo story template for eachphoto story. The embodiment shown in FIG. 4 also illustrates a photostory layout engine 403 for determining an appropriate layout for eachof the photos within each photo story template.

A face recognition engine 402 is included in one embodiment foranalyzing pictures of people and generating facial “signatures” touniquely identify the faces of those people. For example, in oneembodiment, the face recognition engine 402 is executed on all of thephotos in the users contacts database 210, thereby generating facialsignatures for all of the user's contacts and storing the results in thestationery service database 115. The facial signatures may then be usedto identify the contacts in subsequent pictures uploaded by the user405, as described below. Alternatively, the user may simply train thefacial recognition engine 402 by uploading photos and identifying theindividuals shown in each of the photos.

In operation, a user browses to a photo story creation Web page providedby the online photo service 100 and selects a group of photos 400 to beused to create one or more photo stories. Various well known techniquesmay be used for selecting the photos. For example, the web page mayinclude a “browse” button which generates a window for browsing to aparticular folder on the user's computer where the desired photos 400are stored. The user may then highlight multiple photos (e.g., byholding down the SHIFT or CTRL keys while clicking the computer mouse)to select the photos. Alternatively, the user may “click and drag” oneor more photos from a local folder on the user's computer to adesignated region on the Web page to cause the photos to be uploaded. Inone embodiment, a software application (e.g., designed by the onlinephoto service 100) may be installed on the user's computer whichautomatically detects when new photos are imported from a photo sourcesuch as a digital camera and automatically uploads the photos to theonline photo story service 100. In another embodiment, photos areimported from an online photo service such as Flickr, Picasa Web orSmugMug, which have public application programming interfaces (APIs)that allow other web sites to import photos from their servers.

As the photos are uploaded, they are stored within the stationeryservice database 115 (illustrated as Uploaded Photos With Metadata 405in FIG. 4). Virtually all digital photos include some types of metadata,which is used by the various embodiments of the invention describedherein. For example, a photo captured with a digital camera typicallyincludes the date and the time of day that the photo was taken. Inaddition, some digital camera devices may also stamp the photo with a“Geotag” indicating the location where the photo was taken. The locationmay be determined, for example, using Global Positioning System (GPS)technology and stored with each photo. Moreover, some digital cameradevices may be equipped with face recognition hardware/software foridentifying individuals in the pictures (e.g., using the same techniquesas those used by the facial recognition engine 402 on the photoservice). Various other/additional metadata may be associated with eachphoto including, for example, the orientation of the photo (e.g.,portrait, landscape), the photo size and the photo resolution, to name afew.

In addition to the metadata which is generated by the source of eachphoto (e.g., the digital camera), one embodiment of the inventiongenerates additional metadata for each photo, either automatically (byanalyzing the photo) or manually (by requesting information from theuser). For example, as mentioned above, if the user has previouslyexecuted the facial recognition engine 402 and identified people inphotos (e.g., from the pictures in the user's contacts database), thenthe face recognition engine 402 may automatically identify the people ineach of the uploaded photos. Alternatively, the photo stories servicecould provide a user interface to select the names of the people thatare in the photos from the stationery contacts database. In addition,the user may be prompted to enter information for each photo such as aset of keywords identifying the individuals or describing the sceneand/or location presented in each photo.

Regardless of how the metadata is generated, it is stored for each photowithin the set of uploaded photos 405 on the stationery service database115. In one embodiment, the photo story template engine 401 uses themetadata to automatically separate the uploaded photos into differentphoto stories and to select appropriate templates for each photo story.Metadata variables which may be evaluated to separate the photos intodifferent stories may include, but are not limited to, the date and timeon which the photos were captured, the people identified in the photos,and/or the location at which the photos were taken. For example, onetechnique for separating photos into different photo stories is togenerate a separate photo story for each date or date range.Alternatively, or in addition, if people are identified in each photo(either automatically by the face recognition engine 402 or manually bythe user), the photo story template engine 401 may create separate photostories for each identified person or group of people. For example, ifthe user's children are identified, then a photo story may be createdusing only photos of those children. Moreover, a story may be createdfor the location at which the photos were taken (e.g., at home, atwork), or using any combination of date/time, identity of subject,and/or the location of the set of photos.

In one embodiment, the default operation of the photo story templateengine 401 is to generate a different photo story for each date or timerange. However, the user may specify that photo stories should begrouped using different criteria such as the identity of specificindividuals (e.g., particular members of the user's immediate family,specific friends, etc.) or that date, time and identity may be used tocreate the photo stories.

Regardless of how the photos are divided between photo stories, thephoto story template engine 401 then suggests photo story templatedesigns for each photo story based on the attributes of the photo story.The goal is to suggest a design that best suits the story andautomatically arrange the pictures in the design template to reduce thework required by the end user. One embodiment of the template selectionalgorithm initially filters out templates that are not well suited forthe story and then calculates a specific weighting score on eachattribute for each photo story template to create a ranking score foreach template. The templates are then sorted by rank and the top rankedtemplate is used to display the photo story as shown in FIG. 7 c. Theremaining ranked templates are made available to the end user in agraphical user interface (GUI) (see, e.g., FIGS. 7-8) to allow the userto browse through the recommended designs. In particular, in oneembodiment, the photo story template engine selects appropriatetemplates for each story according to one or more of the followingmetadata variables:

Date Taken—The date of the story is used to determine the season or timeof year that it occurred. Each photo story template can include a seasonattribute defined as a period starting from a specific day and month toan ending day and month. For example, a template could be designed forwinter which starts 12/1 and continues until 3/1. If the photo storydate is between those days of the year, the template will be assignedthe “season” score. As another example, a template could be designed fora holiday such as Christmas on 12/25. If the story date is 12/25, seasonscore will be applied to the template.

Number of Photos—The template has an attribute for number of photosdefined as a range from 1 to 100. If the number of photos in the storyis not within the range specified in the template, the template isfiltered out as not applicable for the story. In addition, a “preferred”range of photos may be specified (e.g., 2-5) for each photo storytemplate. If the number of photos in the story falls within thepreferred range, then a relatively higher weight may be applied to theprospective template.

Orientation of the Photo—If a photo is used as the story canvasbackground, the orientation of the photo could impact the design. Atemplate can include an orientation attribute with one of the values ofLandscape, Portrait or Square. This attribute is used to filter outtemplates that do not apply. For example, if the photo is landscapeorientation, all templates that have the orientation attribute set toPortrait or Square and have the photo as background will be filteredout.

Title and Description—The title and description are optional informationthat the user can enter for a photo story. If the story includes eitherof these two attributes, any template that does not include them will befiltered out. The template can also display the date taken and the ageof people in the photo story. If the story includes these attributes andthe template includes the attribute, the template is assigned the scorefor date taken or age, respectively.

Occasion—The design template can include an occasion attribute such asBirthday or Anniversary. If a person in the story has a birthday on thestory date and the template has the birthday occasion attribute set, thetemplate is assigned the Birthday score. Similarly, if a person in thestory has their wedding anniversary on the story date, the template isassigned the Anniversary score.

Template Tags—A template can include keyword tags that describe thedesign. If the story title or description entered by the end usercontains any of the tags in the template, the template is assigned thetitle tag or description tag score. In one embodiment, the title isweighted higher than the description because the title entered by theuser is more important in describing the story.

People Tags—A person defined in a user profile (and possibly storedwithin the contacts database) can include keyword tags that describetheir personality or interests. If the person tags match any of the tagsin the template, the template is assigned the people tag score. Forexample, if a person loves dolls, the user can add a “dolls” tag and atemplate with dolls will be suggested by the recommendation system.

People's Favorite Colors—A favorite color may be specified for a persondefined in a user profile and/or contacts database. If the personidentified in a story has a favorite color that matches a color in thetemplate, the template is assigned the favorite color score.

Location—A template can have a location, region, and/or point ofinterest attribute that defines the geographic area and/or the specificlocation for which the design is best suited. In one embodiment, aregion is defined as a city, state, or group of states. If the locationof the photo story is known (either as part of the photo meta data ormanually entered by the user) and the template has the same location orregion, the template is assigned the location or region score/weight.For example, a template could be designed for the Monterey Bay Aquariumwith sea creatures and logos from the aquarium. If a story has thelocation set to the Monterey Bay Aquarium, the template is assigned thelocation score. As another example, if a photo was taken in New YorkCity and a template is set with New York City as the region, then thetemplate is assigned the region score.

Design Style—A template can include a design style attribute such as“Artistic,” “Classic,” “Vintage,” “Whimsy” or “Contemporary.” A userprofile or contact entry within the user's contacts database can includea design style attribute either selected by the user or based on pastphoto story usage and orders. If the design style attribute for the usermatches the design style attribute of the template, the template isassigned the style score.

Mood—The mood of the photo story can also reflect on the designtemplate. By way of example, mood options may include “Angry,” “Happy,”“Relaxed,” and “Sad.” As with other variables, the preferences of theuser and/or people identified in the photos may be determined to selecta mood.

Relationships—A template can include relationships that define thepersonal relationships for which the design is best suited. The userprofile can include a contacts database with people and theirrelationship to the user (e.g., son, daughter, wife, mother, father,etc.). Consequently, when people are identified in photos, the photostories will include attributes associated with the people in the story,which then associates the story to relationships. If the story includespeople with relationships that match the relationships set on the photostory template, then the template is assigned the relationship score.The template can also include a relationship group such as “My Family”or “Couple”. If a relationship group is set on the template, it will befiltered out if the story does not include the exact relationships. Forexample, for the “My Family” relationship group, the story must includethe relationship defined in the user profile as “my family.” The“Couple” relationship group may be defined as husband and wife.

In one embodiment, the following weights are applied to attributes bythe photo story template engine 401. The attributes are weighted byimportance when determining the “best matches” for the photo story withhigher values meaning higher importance:

  Season Score = 10 Location Score = 50 Region Score = 10 RelationshipScore = 50 Birthday Score = 200 Anniversary Score = 100 Description TagScore = 10 Title Tag Score = 25 People Tag Score = 20 Style Score = 10Mood Score = 10It should be noted, however, that the underlying principles of theinvention are not limited to any particular set of attribute weights.

In one embodiment, the photo story template engine 401 scans through allof a user's photos 405 (or an identified subset of the user's photos) tocreate new stories based on requirements specified by the user. Forexample, the user may specify that a story is to be created using photosof the user's children. In response, the photo story template engine 401will identify all photos of the user's children (e.g., using themetadata discussed above) and generate photo stories using these photos.Similarly, the user may specify that the photo story template engine 401should generate a photo story for the holiday season for a particularyear, for example, by specifying a date range such as Dec. 24, 2009 toJan. 1, 2010. The photo story template engine 401 then generates storiesusing pictures from within the specified date ranges. One embodiment ofthe service will provide the user with pre-configured options forcertain holidays (e.g., “Thanksgiving '09 Photo Story”). Because eachphoto has metadata associated with it, virtually any user query directedto the metadata may be applied to identify a subset of photos andgenerate a photo story based on the query results. Of course, theunderlying principles of the invention are not limited to any particulartype of metadata or query.

In one embodiment, after a set of prospective photo story templates areselected and ranked by the photo story template engine, the photo storylayout engine 403 automatically generates an exemplary photo storylayout using the selected photo story templates, photos, and textassociated with each photo story. In one embodiment, each photo storytemplate includes photo areas and text boxes based on relativedimensions and coordinates within the canvas of the photo story designtemplate. The photos in the story are arranged within the photo areasand the size and coordinates of each photo area are dynamically adjustedby the photo story layout engine based on the number and size of thephotos.

In one embodiment, each photo area within a photo story design templateincludes an attribute for a minimum and maximum number of photos and thesize and position of the photo area is determined based on the numberand size of the photos in the story. One embodiment of the photo storylayout engine 403 uses the following attributes to arrange the photoswithin a photo template:

zIndex—The design elements (e.g., photo areas, text boxes, graphics) arelayered on the canvas based on the zIndex with larger valuesrepresenting higher layer in the z-plane.

Layout Order—For normal layout flow, the design elements are positionedbased on the layout order starting from top-left and going tobottom-right.

Float—Each design element may be assigned a float attribute to floatleft or right on the canvas.

Position Coordinates—The design elements may have values for left, top,right and bottom defined as percentages of the width or height of thecanvas.

Sort Order—The photo areas have a sort order attribute that defines theorder that photos are arranged in the photo areas.

Maximum Photos—This attribute defines the maximum number of photos thata particular photo area can contain.

Dimensions—The design element has a size defined as width and height asa percentage of the width and height of the canvas.

Effects—A photo area can includes effects including rotate, feather andcolor conversion (black and white, sepia). The specified effects areapplied to any photos within the designated photo area (e.g., everyphoto is feathered by a specified percentage).

Border—A photo area can include a border attribute to add a border withwidth, padding and color to each of the photos within that photo area.

Photo Sizing—This attribute defines how the photos in the photo areawill be cropped to fit within the photo area with options includingbest-fit, uniform width and uniform height.

Photo Alignment—This attribute defines how the photos are aligned anddistributed within the photo area. Values are align-left, align-center,align-right, align-top, align-middle, align-bottom,distribute-horizontally and distribute-vertically.

Text Attributes—Text boxes have attributes for font, color, size,leading, alignment, and no-wrap. If the no-wrap attribute is set, thetext size is reduced to make the text fit within the text box withoutwrapping to the next line.

Text Variables—Text boxes can include text variables that will beassigned values from the photo story information. These variablesinclude Title, Date Taken, Location, Description, Family Name, MemberName, Member Home Address, People Names, People Ages, Birthday Age, andAnniversary Years.

The photos within a photo area are arranged based on an algorithm usingthe photo area attributes described above. The dimensions of the photoarea are first determined based on the size of the template anddimensions attributes set on the photo area and the number and size ofthe photos in the photo area. For example, if the minWidth and maxWidthare both 100%, minHeight is 10% and maxHeight is 30%, and there arethree photos in the photo area, the photo area width is set to the widthof the template canvas and the height is determined by the height of thephotos.

FIG. 5 a illustrates a method for generating and sharing photo storiesin accordance with one embodiment of the invention. At 501, new photosare received. For example, the end user may select a new set of photosto upload to the online photo service 100. At 502, metadata is extractedfrom the photos, manually entered by the end user and/or generatedautomatically by the photo service 100. For example, the date/time maybe read from each of the photos, the user may manually enter adescription of the content of the photos, and/or the photo service mayautomatically analyze the photos to generate metadata (e.g., extractingcolors from the photos or using the facial recognition engine 402).

At 503, user input is received to filter and/or arrange the photos basedon specified criteria. For example, the user may specify that aparticular photo story is to be focused on a particular subject (e.g.,the user's children) during a particular day (e.g., Halloween). The usermay also adjust the time range to group more or less photos together. At504, the photo story template engine automatically separates the photosinto photo stories based on the collected metadata and/or user input. At506, the user may adjust the photo stories by, for example, removingphotos, adding photos and/or filtering out certain photos (e.g., usingmetadata as the filtering criteria). At 507, when the photo stories havebeen completed, the user shares and/or prints the photos 507.

In one embodiment, the user is provided with the option of specifyingone or more social networking sites through which the photo stories maybe automatically shared (e.g., Facebook, Twitter, etc.). Once the user'sauthentication data for these social networking sites has been enteredand stored on the photo service 100 (e.g., the user's ID code andpassword), a social networking interface 450 can automatically connectto one or more external social networking sites 451 on behalf of the enduser to upload the new photo stories. Thus, in one embodiment, at theend of the photo story creation process, a “Share Photo Stories”hyperlink is provided on the photo stories Web page to provide thisfunctionality. In response to the user selecting the “Share PhotoStories” link, the social networking interface 450 will thenautomatically connect to the external social networking sites 451 withthe user's authentication data and upload the new photo stories to thesocial networking sites 451. As is known in the art social networkingsites such as Facebook expose a Web services API to allow these types oftransactions.

In addition to the “Share Photo Stories” link, in one embodiment, a“Print Photo Stories” link is provided. In response to selecting the“Print Photo Stories” link, the completed photo story 406 is sent as anew print job 255 and printed by an internal printer 251 or using anexternal print service. In one embodiment, the user may specify a groupof individuals to always receive printouts of new photo stories (e.g.,the user's parents, grandparents, siblings, etc). The online photosservice 100 maintains the identity of these individuals (e.g., withinthe contacts database 210) so that they can be readily identified inresponse to the user selecting “Print Photo Stories.”

FIG. 5 b provides additional details associated with how the photo storytemplate engine selects a set of photo story templates (step 505 in FIG.5 a). At 511, the photo story template engine analyzes the metadataand/or user input to separate the photos into photo story groups. In asimple embodiment, the photos are separated into photo story groupsbased on the date/time that the photos were taken. Alternatively, or inaddition, other variables such as the identity of the photo subjects maybe used to group the photos.

At 512 photo story design templates are filtered based on the photostory metadata and/or user preferences. As mentioned above, any of thevariables associated with the photo story such as the number of photos,the orientation of the photos, the existence of a title/description,etc, may be used to filter out unsuitable photo story design templatesfrom consideration. For example, if the number of photos in the photostory is not within the range specified in the template, the template isfiltered out as not applicable for the photo story.

At 513, the existing photo story design templates are ranked based onthe metadata and/or user input/preferences. As mentioned above, eachphoto story design template is assigned a weight based on certainvariables. If those variables are detected in the group of photo storyphotos, then the weight is applied to the design template. For example,if a particular set of photos fall within a specified date range, thenthe season score of 10 may be applied to all photo design templatesassociated with that season. Similarly, if the photos are associatedwith a particular location, then the location score of 50 may be appliedto all photo story design templates associated with that location. Whenall applicable weights have been applied, the weights are added toarrive at a “score” for each prospective photo story design template.Finally, at 514, the photo story design templates are provided to theend user based on the calculated rankings. In one embodiment, the photostory design template having the highest ranking is selected andpopulated with photos by the photo story layout engine 403. The user maythen have the option to peruse the other photo story design templates.Upon selection of each design template, the photo story layout engine403 will automatically populate the deign template with photos (e.g.,using the layout techniques described herein).

In addition, in one embodiment, the photo story layout engine 403extracts the main colors from the story photos and are used toautomatically color certain design elements within the photo storydesign template. This results in a layout and design that complementsthe photos without overpowering the photos. One embodiment of the designcoloring algorithm uses the colors from the photos in the story andreplaces the colors in certain specified design elements with thesecolors. The colors are extracted by using image processing functions toreduce the number of colors to the 16 most dominant colors in the image.More specifically, in one embodiment, dithering is applied followed byquantization to reduce the number of colors to 16 in order to smooth outthe color. This results in colors that the human eye perceives acrossthe image as a whole instead of the extreme peaks in the colors. Thephoto colors are stored as RGB (red, green, blue) values with a usagevalue that is set equal to the number of pixels within the quantizedimage that are that color. The 16 colors for a photo story aredetermined by performing a union of the colors from all of the photos inthe story and then sorting by the usage values to arrive at the 16 mostused colors. The design elements such as overlay frames and designaccents (embellishments) are then colored by replacing specific colorswith one or more of the most used colors from the photographs. Thecolors to be replaced are determined by extracting the colors from theoverlay image (such as a PNG image or PDF vector graphics file). Thetemplate designer selects which colors should be replaced with colorsfrom the photographs. This is shown in the overlay form in FIG. 14. Forexample, the colors shown in 1407 are the colors (defined as RGB hexvalues) that the designer has selected to be replaced.

FIG. 6 illustrates one embodiment of an algorithm for selecting colorsfor design elements based on the main detected colors within the storyphotos. At 521, each of the images within the photo story are quantizedto reduce the image to a specified number N of colors. In one embodimentN=16; however, the underlying principles of the invention are notlimited to any particular number of quantized colors. At 522, usagevalues are generated for each color by (in one embodiment) adding thenumber of pixels within the quantized image that are each color. At 523,the results from each of the photos are combined to arrive at usagevalues for the photo story. At 524 the colors are sorted based on usagevalues and, at 525, colors of one or more design elements are selectedbased on the sorted usage values. For example, the design elements suchas overlay frames and design accents (embellishments) may be colored byreplacing specific colors with one or more of the most used colors fromthe photographs. The colors to be replaced are determined by extractingthe colors from the overlay image (such as a PNG image or PDF vectorgraphics file).

One embodiment of a graphical user interface (“GUI”) for viewing andmanipulating photo books created by the photo story template engine 401and photo story layout engine 403 is illustrated in FIGS. 7 a-c and FIG.8. Each photo story is provided with its own regions 701, 702, 801 inwhich the user can modify the photo story in various ways and ultimatelyshare the photo story with other users. As mentioned above, in oneembodiment, the photos are separated into the different photo stories bythe photo story template engine 401 (e.g., based on metadata such asdate/time, photo subject, location, etc). A preview image for each photostory is initially displayed with photos integrated within thehighest-ranked design template selected by the photo story templateengine 401. In FIGS. 7 a-b, this includes photo story preview images 705and 706 and in FIG. 8, photo story preview image 805. The photosselected for use in each photo story are also displayed within theirrespective photo story regions. For example, a single photo 708 isdisplayed within photo story region 701; four photos 710 are displayedwithin photo story region 702; and four photos 810 are displayed withinphoto story region 802.

New stories may be added via an “add a story” link 731 and photos may beadded via an “add photos” link 730. In one embodiment, photos may bemoved between photo story regions 701-702, 801 by clicking and draggingthe photos with a mouse or other cursor control device. Moreover, newphotos may be added to the photo story regions by selecting them from afolder on the user's computer system and copying/pasting or clicking anddragging them to the photo region. When a new photo is moved to thephoto region, the same analysis as described above may be employed(e.g., to identify the individuals within the photos).

In addition, in one embodiment, photos may be rearranged within thephoto story previews 705-706, 805 by modifying the order of the photoswithin a photo story region. For example, to change the large photo atthe bottom of the photo story preview 706 shown in region 702, the usermay click and drag the leftmost photo within the group towards theright, thereby causing the order of the photos to be rearranged withinthe photo story preview 706. Said another way, the order of the photoswithin the photo story corresponds to the order of the photos within thephoto region 702.

Within each photo story region, a data entry field 720 is provided forentering a photo story title; a date/time field 721 is provided for thedate/time during which the photos were taken; a location field 722 isprovided to indicate the location at which the photos were taken; and adescription field 723 is provided, allowing the user to enter adescription for the photo story. In one embodiment, some of the fields,such as the date/time field 721 are automatically entered by the photostory template engine 401 while other fields are entered manually by theuser. In one embodiment, before the user has entered any data to thesefields, the title field 720 reads “untitled;” the location field 722reads “add a location” and the description field 723 reads “add adescription.”

When a user changes the number of photos within a photo story (e.g., byadding or deleting from one of the photo regions 701, 702, 801), thephoto story template engine 401 may re-rank the photo story designtemplates based on the change and/or the photo story layout engine 403may adjust the layout of the photos within the current photo storydesign template. In FIG. 7 b, a photo has been removed from photo storyregion 702, thereby changing the number of photos from 4 to 3. Inresponse, the photo story layout engine 403 automatically rearranges thephotos within the photo story design template 706, as shown.

FIG. 7 c illustrates the stationery design template 706 in greaterdetail. This particular photo story design template includes two photoregions, 780 and 782, and one text region 781. The first photo region780 is configured to hold from 1-4 photos and the second photo region782 is configured to hold a single photo. Thus, in one embodiment, thephoto story layout engine 403 sizes the photo within the second photoregion 782 to be the same width as the photo design template (minus thesize of the border if one is used). To do so, it may crop the photovertically and/or horizontally to generate the best fit based on thevertical size constraints specified in the photo story design templatefor this photo region. In one embodiment, if the photo story layoutengine 403 scales a photo it keeps the horizontal and vertical scalingproportional (to ensure that the image is not visually compressedhorizontally or vertically). Similarly, the photo story layout enginemay adjust the size of the text region 781 based on the size of thephotos used in the two photo regions and the text region constraintsspecified within the photo story design template. With respect to thefirst photo region 780, the width of the photo story design is splitevenly between each of the photos. Thus, because the horizontal andvertical scaling must remain consistent, the first region is shallowerin the vertical orientation with three photos than with two (as shown bya comparison the design template 706 of FIGS. 7 a and 7 b).

In one embodiment, a link is provided within each photo region 701, 702,801 to allow the user to toggle through each of the ranked photo storydesign templates. In response, the photo story layout engine 403 willautomatically layout the photos and text within each selected photostory design template.

In one embodiment, selecting the “Next: Preview” link 707 shown in FIG.7 a generates a finalized preview of the photo story 705 such as thatshown in FIG. 7 c and may also provide options for viewing other photostory templates. As illustrated in FIG. 7 c, the preview Web page mayinclude links 751 and 752 to share the photo story on a socialnetworking site or via email, respectively. An edit link 753 allows theuser to edit any of the photo or text boxes within the photo storytemplate (e.g., providing features such as resizing, moving, cropping,etc) within the size/coordinate constraints set by the photo storydesign template. A “try again” link 754 toggles between different photostory design templates. Finally, a share link 755 provides a pluralityof different options for sharing the photo story design template (e.g.,different social networking sites, different email addresses, differentprint options, etc.) such as those shown in FIG. 7 e (described below).In one embodiment, if the current photo story is not the last to bereviewed in succession, then the share link 755 is simply a “next” link,allowing the user to move to the next photo story preview.

In one embodiment, the text used within text regions of the photo storytemplates may include variables which are adjusted based on the subjectin the photo. For example, if the subject has been identified and thebirthdate of the subject is known (e.g., from the contacts database),then the age of the subject may be generated as text on the front of thephoto story template. For example, in FIG. 7 c, the subject has beenidentified (“Megan”) and the age of the subject has been calculated asof the date that the photos within the photo story design template weretaken (“9 months” in the example). The use of variables within textboxes is described in detail in the co-pending applications such asSYSTEM AND METHOD FOR DESIGNING AND GENERATING ONLINE STATIONERY, Ser.No. 12/188,721, filed Aug. 8, 2008, referenced above.

FIG. 7 d illustrates one embodiment of a photo story GUI in which twosets of photos have been separated into two different photo story groups730, 731. In this example, the photos groups have not yet been assignedphoto story templates. At this stage, the user may opt to delete photosfrom each of the groups 730-731 and/or manually add photos to thegroups. Each of the photo groups are provided with a title 740, adate/time 741, a location 742 and a description 743. As mentioned above,some of this information may be determined automatically (e.g., usingmetadata such as time/date) and all of this information is editable bythe user. Once the user has edited the photos and the informationrelated to the photos, selecting a “create story” link causes the photostory template engine 401 to select an appropriate template and thephoto story layout engine 403 to layout the photos within the template(as described above). The template then appears with the photos withinthe photo story region (e.g., as shown in FIG. 7 a-c). An update storylink 745 causes the photo story to be updated with newly added photos,deleted photos, new photo arrangements, and/or new information enteredby the user. A delete story link 746 deletes the photo story.

FIG. 7 e illustrates a share and print GUI which provides a set ofuser-selectable options 751-756 for sharing and/or printing photostories. In one embodiment, this GUI is generated in response to theuser selecting the share link 755 shown in FIG. 7 c. The photo storiesto be shared are displayed within a photo story region 750 and links toskip sharing/printing 760 and to share/print 759 are provided. Theoptions for sharing/printing may have been pre-specified by the end userand, in FIG. 7 e, include well known websites such as Facebook 751,Twitter 752, Picasa 753, and Flickr 754. In addition, the options mayinclude email 755 (for emailing copies of the photo stories) and prints756 (for printing paper copies of the photo stories). In one embodiment,the recipients to receive the email and prints have been specified bythe user and stored in the user's personal data on the online stationeryservice.

As shown in FIG. 8, in one embodiment, a listing of people 802 may begenerated within a photo story region to show who is pictured in thephoto story. The listing 802 may be generated, for example, in responseto the user selecting the “people” link in FIG. 7 a. A graphic isprovided next to the name of each person in the list which, whenselected, removes that person from the list and removes any photosincluding that person from the photo story region 802.

FIG. 9 illustrates one embodiment of a graphical user interface formanaging and browsing existing photo stories (e.g., in response toselecting a “My Photo Stories” link). One or more photo stories aredisplayed within a photo stories region 911. Particular groups of photostories are displayed within the photo stories region 911 based onselections made by the user within a set of filtering options 901-905.For example, a graphical timeline 901 is provided at the top of the GUI.Upon selection of a particular date or date range (e.g., month, year)within the timeline, photos occurring during that date range aredisplayed within the stories region 911. A scroll graphic 910 is alsoprovided allowing the user to scroll through the timeline, therebycausing new sets of photo stories to be displayed as the scroll graphicis scrolled.

A separate set of filtering options is provided to the left includingoptions for filtering photo stories based on the time 902, options forshowing photo stories involving specific people 903, specific places 904and recently added photo stories 905. As filtering options are selectedat the left, an indication of the filtering appears within the headingof the GUI (“ALL (128)” is shown in the example). In one embodiment,filtering options may be combined. For example, the user may select twodifferent individuals under “people.” In response, the photo story GUIwill only display photo stories having both of the selected people assubjects (i.e., the people are ANDed together). In addition, in oneembodiment, once a particular person is selected, a list of selectabletags are generated allowing the user to browse through all of thestories that the selected person is in by selecting the different tags(e.g., birthday, hat, cars, park, etc).

In one embodiment, each user is required to establish an account on theonline photo service 100. FIG. 10 illustrates one embodiment of agraphical user interface (GUI) for editing a user profile with datafields for the users first and last name 1001 and 1001, respectively,the user's nickname 1003, the user's family name 1004, the userspreferred design style 1005 (e.g., classic, modern, etc.), the user'sinterests 1006, a photo of the user 1007 (including a browse button tolocate the photo on a client computer), the user's phone number 1008,email address 1009 and address information 1010. Other members of theuser's friends and family may enter similar profile information and mayshare this information with the end user. Alternatively, or in addition,as mentioned above and in the co-pending applications, the user maystore similar information for each of the contacts within the user'scontacts database.

FIG. 11 illustrates one embodiment of a GUI for browsing individualswithin the photo stories contacts database 210. A top row of alphabeticcharacters 1101 allows the user to jump to a certain point within thedatabase corresponding to a selected character. A search field 1102allows the user to search for specific contacts and a “new person” fieldallows takes the user to a new person details GUI (such as that shown inFIG. 12). The GUI provides a listing of individuals 1105 in the contactsdatabase, organized alphabetically. Also shown are certain pertinentdetails 1104 related to the person such as a relationship to the user(e.g., daughter, son) a birthday and/or an anniversary.

FIG. 12 illustrates a “person details” page for creating and/or editingdetails associated with a particular contact within the photo contactsdatabase 210. A name is specified in field 1201; a nick name in field1202; a birth date in field 1203; an anniversary date in field 1204; arelationship to the user in field 1205; and one or more tags related tothe person in field 1206. Finally, in field 1207, the user may select aphoto of the person (e.g., by browsing to a particular folder andselecting a photo). As described above, the photo may be used toidentify the person in photos uploaded to the online photo service 100.For example, the facial recognition engine 402 may employ facialrecognition techniques to register a facial recognition signature of theperson. This signature may subsequently be used by the facialrecognition engine 402 to identify the user in uploaded photographs.Finally, a submit button 1208 is provided for saving the person detailswithin the photo contacts database.

FIG. 13 illustrates one embodiment of GUI for creating and/or editing aphoto story template. The GUI includes a personalize link 1320 which, inone embodiment, takes the user to the photo area and text box GUIsillustrated in FIGS. 15 and 16, respectively; and an overlays link 1321which takes the user to the overlays GUI shown in FIG. 14. The templateGUI includes a name field 1301 for entering a template name and abackground selection graphic 1302 for either specifying either a coloror a picture to be used as a background image for the photo storytemplate. A browse button is provided to browse through folders on theuser's computer to select a photo.

An orientation menu 1303 provides options for selecting the photo storysize and shape. Examples may include square (e.g., 12″×12″, 18″×18″,etc), rectangular (e.g., 12″×18″, 18″×24″, etc.), and any othersizes/shapes including both industry standard sizes/shapes andnon-standard sizes/shapes. Drop down menu 1304 provides for theselection of a particular theme (e.g., humorous, sentimental, religious,etc.); drop-down menu 1305 provides for the selection of a particularstyle (e.g., modern, classical, whimsy, etc.); drop-down menu 1306provides for the selection of a particular type of occasion (e.g.,birthday, anniversary, wedding, Thanksgiving, etc.); and selection list1307 provides options for specifying relationships to be associated withthe photo story template (e.g., family, couple, parents, daughter,etc.). Drop down menu 1307 provides options for selecting a particularregion (e.g., California, San Francisco, New York, Grand Canyon, etc)and month/day drop-down menus 1309 provide for the selection of aparticular date range and/or season. Selection boxes 1311 indicatewhether text specified in the photo story regions 701-702 describedabove will be utilized in the photo story design template. Theillustrated options include title, date, description and age (of thesubjects). Finally, the “tags” data entry field 1312 is used to enterone or more tags to categorize the photo story template (e.g., so thatthe photo story template may be identified by matching a user'sdescription of the photos in a photo story with the tags).

In one embodiment, selecting the overlays link 1321 brings a user to theoverlays GUI shown in FIG. 14, which allows the user to configure anoverlay for the photo story template. In one embodiment, an “overlay”can be used to provide various different graphical features surroundingthe pictures of the photo story. Each overlay is comprised of one ormore overlay elements. In FIG. 14, a data field is provided for theoverlay element name 1401, a zIndex—a value which specifies the order inwhich the overlay element will be placed on the photo story designcanvas. As is known in the art, a zIndex value is a Cascading StyleSheets (“CSS”) property often used in Web page design that sets thestack order of a specific set of elements. An element with larger zIndexis always placed in front of an element with lower stack order. Dataentry elements 1403 allow the end user to select a color overlay and/oran image file to be used as an overlay. A browse button is provided toallow the user to search through folders for the image/color file. Theposition of the overlay within the photo story template is specified indata entry fields 1404. In one embodiment, the position is specified inunits such as inches or pixels and defines the boundaries of the overlay(left, top right, bottom). Data entry fields 1405 are used to specifyand absolute or relative width and height of the overlay within thephoto story template (e.g., as a percentage of the size of the photostory template canvas). In one embodiment, setting only one of these twovales ensures that the aspect ratio will be maintained (i.e., the otherwill be set automatically while keeping the aspect ratio consistent). Indata entry field 1406, the transparency value is set using a value of 0(not transparent) to 100 (completely transparent). A set of color valuesare specified in fields 1407. In one embodiment, one or more of thecolor values are selected based on the main colors used in thephotographs of the photo story (as described above). As previouslydescribed, the colors in 1407 are in RGB hex format. In one embodimentthe colors are replaced by colors in the photos. The default values whenthe user uploads the overlay image is the main colors from the overlayimage, but they can be removed or replaced with values entered by theuser. Drop-down menus for selecting a theme; a style 1409; and a region1410 are provided and month/day drop-down menus 1311 provide for theselection of a particular date range and/or season.

As illustrated in FIG. 15, a photo area template includes a data fieldfor a template name 1501 and a zIndex value 1502. As described above,the zIndex value specifies the order in which the photo area is laid outon the photo story canvas in relation to other design elements (e.g.,other photo areas, text boxes, overlays, etc). A layout order drop-downmenu 1503 specifies the order in which the different photo areas will belaid out on the photo story canvas (e.g., using an integer value). A setof position drop-down menus and data fields 1504 allow the user tospecify the position of the photo area. The user may specify positionusing prearranged selections within the drop-down menus (e.g., floatleft, float right) and by specifying left, top, right, bottom values aspercentages of the width or height of the photo story canvas. Theleftmost drop-down menu shown in 1504 provides options for normal flow,absolute and relative. In one embodiment, Normal Flow is the layout flowthat starts at top-left and goes to bottom-right, Absolute meansabsolute coordinates are used, and Relative means offset from theposition that the element would be positioned by Normal Flow. In oneembodiment, these properties are used in cascading style sheet (CSS)positioning so that one of ordinary skill in the art familiar with CSSwould understand how to configured the design. A sort order valueselected via a drop-down menu 1505 specifies the order in which thephotos will be placed into the photo areas. For example, when there aremultiple photos in the photo story, they are sorted with photo number 1on the left and increasing to the right. The sort order in the photoarea specifies in which order the photos will be layed out when thereare multiple photo areas in the template. For example, with one photoarea with maximum photos set to 1 and sort order 1 and another withmaximum photos set to 4 and sort order 2, the first photo in the storywill go in the first photo area and the 2 through remaining photos willgo in the second photo area. A maximum photos drop-down menu 1506 allowsthe user to specify the maximum number of photos for the photo area anda set of data fields 1507 allow the designer to specify minimum andmaximum width and height for the photo region. Data fields 1508 areprovided for specifying different photo effects such as rotating andfeathering (e.g., using a percentage or absolute value) and for settinga border around the photo region 1509 (specifying width, padding amountand color). A photo sizing drop-down 1510 defines how the photos in thephoto area will be cropped to fit within the photo area with optionsincluding best-fit, uniform width and uniform height and a photoalignment drop-down 1511 defines how the photos are aligned anddistributed within the photo area. Values include align-left,align-center, align-right, align-top, align-middle, align-bottom,distribute-horizontally and distribute-vertically. In one embodiment,the foregoing variables are interpreted by the photo story layout engine403 when determining how to arrange the photos on each photo story.

As illustrated in FIG. 16, a text box template includes data fields fora text box name 1601; a zindex value 1602; a layout order 1603; aposition 1604; and text box dimensions 1605 (specified either as apercentage of the photo story template or using absolute values).Drop-down menus provide options for font type 1606, color 1607, size1608 and leading 1609 and drop-down menus 1610 allow the user to specifyalignment values for the text box. Finally a data entry field 1610allows a designer to specify default text to be used in the text box. Inone embodiment, variables may be used as described above. For example,an AGE variable may be included to specify the age of a particularsubject. These variables are automatically populated using informationfrom the user's contacts database (once the subjects in the photos areidentified).

System and Method for Photo Story Theme Selection

While online photo books are replacing the traditional photo album asthe best way to showcase memories in photos, the user still has toinitiate the process of creating a book based on an occasion or storythey want to tell. Most consumers never think to create a photo book orthey are intimidated by the process and fear that it will require toomuch time to finish. In addition, the user still has to determine how tocreate stories from their photos or arrange them in pages based onevents or the date they were taken. In many cases, a photo book is moreinteresting when the photos are arranged to tell a story instead of arandom collection of photos. This is a problem for many consumersbecause they are not good at creating stories. One embodiment of thephoto storybook service solves these two problems by automaticallyproviding suggestions for storybook themes and using storybook templatesto make it easier to tell a story with photos.

In one embodiment, the automatic storybook theme suggestions are derivedfrom the common occasions in many people's lives based on their age andfamily status. In addition, many people want a yearbook of memories foreach year and a storybook for each year of their children's lives. Inone embodiment, the photo storybook service uses a customer's profilecontaining birth date, gender and family members to suggest storybookthemes at different times throughout the year. The storybook themes arealso available for browsing and manual selection by the end user.

The storybook templates are designed by storytellers to simplify thecreation of stories with the user's photos. After choosing a storybooktheme, the user is provided a selection of storybook templates to choosefrom. The service provides suggestions on which templates are mostapplicable by analyzing all the metadata of the photos to determine thedistribution of photos taken over time, what people are in the photos,where they were taken, and tags or descriptions that specify the topicof the photos. Each storybook template includes design elements, pagelayouts, and instructions that guide the user to select their photos tofit the story. As the user adds photos to the storybook they areautomatically filled into the pages of the book. After completing thestorybook template, the user

As illustrated in FIG. 17, one embodiment of the photo service 100includes a photo story theme engine 1701 for collecting user input andresponsively providing recommendations for a photo story “theme” 1702based on the user's specifications. For example, in response to the userselecting a time period of a full year for the storybook (e.g., all of2010), the photo story theme engine 1701 may select one or more“yearbook” theme options. In response to the user selecting a particularevent (e.g., a birthday), the photo story theme engine 1701 may select atheme which is specifically adapted for that event (e.g., a birthdaytheme). As indicated in FIG. 17, each theme may be associated with oneor more photo story templates 404.

In operation, as illustrated in FIG. 18, the photo story theme engine1701 initially determines user preferences and collects user data neededfor selecting one or more themes at 1801. For example, the user mayselect a plurality of photos to be used in the storybook ranging acrossall of the months in a particular year (e.g., all of 2010). As such, at1802, the photo story theme selection engine 1701 may provide a yearbooktheme as one option from which the user may select. Similarly, if theuser selects photos all from a particular date, then the photo servicemay determine whether that day corresponds to a birthday of the user ora member of the user's family (e.g., by querying the user's contactsdatabase). In such a case, at 1802, the photo story selection engine1701 may provide a birthday theme as one option from which the user mayselect. Various other use cases are contemplated including parties andevents, life events, vacations, seasonal/holidays, and/orhobbies/interests. In one embodiment, the photo story theme engine 1701selects the storybook theme based on information in the customer profileand the time of the year. The user selecting photos as described is oneparticular example, but other use cases are also contemplated. Forexample, a yearbook theme may be suggested by the photo story themeengine 1701 near the end of a year. A children yearbook or birthdaystorybook may be suggested on or around a birthday in the user's friendsdatabase. A holiday themed book might be suggested around a holiday likeChristmas. In one embodiment, the service has access to the user'sphotos so that when the user adds new photos, the photo story themeengine triggers a suggestion to make a storybook based on the datetaken, the number of photos, and metadata information (e.g., the user'sdescription).

As another example, in one embodiment, the photo storybook is built overtime following a particular story book template to help the user capturemoments and tell his/her story. For example, the user might start ayearbook in the beginning of the year by selecting a yearbook theme andtemplate. The service would then send periodic emails (or otherreminders) asking the user to capture pictures for the storybook or addpictures to the storybook.

In one embodiment, rather than deducing the particular theme based onthe time period over which photos are selected, the photo story themeselection engine 1701 may simply ask the user a series of questions todetermine the theme (e.g., “Is this story book being created for aparticular event?”).

For example, in one embodiment, the user is presented with a list ofstorybook themes or occasions organized by category. The photo storytheme engine 1701 may also include a search feature to allow the user tofind a storybook template based on keywords. By way of example, and notlimitation, in one embodiment, the following hierarchical categories areprovided for user selection:

1. Wedding a. Engagement b. Bachelor Party c. Bachelorette Party d.Bridal Shower e. Wedding f. Honeymoon 2. Baby a. Baby Shower b. New BabyArrival c. Baby First Month d. Baby First Year e. Pregnancy 3. Kids a.Birthday Party b. Children Yearbook 4. Yearbooks a. Children Yearbook b.Family Yearbook c. Friends Yearbook d. School Yearbook 5. Parties/Eventsa. Baby Shower b. Bachelor Party c. Bachelorette Party d. Birthday Partye. Bridal Shower f. College Reunion g. Family Reunion h. Friends Reunioni. High School Reunion j. Party k. Wedding 6. Vacations a. CruiseVacation b. Honeymoon c. Road trip d. Travel 7. Seasonal a. Christmas b.Father's Day c. Halloween d. Mother's Day 8. Interests a. Animals b.Fishing c. Hobbies d. Our Home e. Pets f. Play g. Scenes h. Sports i.Travel

After the user and/or the story book theme engine 1701 selects astorybook theme or occasion, the user is provided a list of one or morestorybook templates for the selected theme. For example, for the “Baby'sFirst Year” theme, the story book theme engine 1701 might provide thefollowing templates:

-   -   1. Classic Baby—A timeless story to introduce a baby to the        world, starting with pregnancy and going through the first few        weeks.    -   2. I Can—A simple and cute way to showcase your baby's        milestones in pictures including “I Can Sleep”, “I Can Eat”, and        “I Can Walk”.    -   3. Baby Body Book—Showcase your baby through the classic baby        body book with your pictures of your baby's body parts and        faces.

In one embodiment, once a particular theme template has been selected,at 1803 the photo story theme engine 1701 steps the user through theprocess of selecting photos and entering other data needed to completethe story book theme template. For example, a storybook template caninclude text variables that are populated with values entered by theuser (e.g., storybook name, the year, individuals shown, honoree, eventdate, etc.).

In one embodiment, a storybook template comprises one or more “stories”each having a name and questions or instructions to guide the user inselecting the photos to fit the story. For example, the “Baby's FirstYear” theme template mentioned above might include a story of “ExpectingMommy” for which the user may be instructed to select pictures of theexpecting new mother. In this way, the user is guided through theprocess of selecting appropriate content for the selected story booktheme.

In one embodiment, the photo service 100 maintains a database 115 of theuser's photos. In addition, the photo service 100 may connect to one ormore external photo services 1706 via a photo service interface 1705.For example, the interface 1705 may provide authentication data requiredto connect to the user's account on the external photo service 1706 byutilizing a public API exposed by the external photo service 1706. Inaddition, the photo service 100 may maintain a listing and metadata ofphotos stored on the user's client computer 140. Regardless of where thephotos are stored, the photo service 100 may display the photos for theuser in chronological order based on the date taken. In one embodiment,the photo story theme engine 1701 allows the user to filter the photosto those that are relevant to the particular selected theme (e.g.,tagged with the Honoree). The user may select photos for the story byclicking on the thumbnail or dragging the photo to the story area. Asset forth in detail below, the user interface may indicate which photoshave been previously selected for inclusion in the storybook.

Once the user has selected photos and other data required to completeeach individual story, at 1804, the story book theme engine provides theuser with a finalized story book. The user may then choose to purchasethe story book and/or to distribute the story book to one or more otherusers.

The following use cases and features are provided in one embodiment ofthe invention. Some of these features have already been discussed above.It should be noted, however, that these specific features are notrequired for complying with the underlying principles of the invention.

-   -   1. Use Cases. One embodiment of the photo storybook engine 1701        supports the following use cases. In general, the system is        applicable to any large collection of photos to guide the user        to organize the photos into a story that could be distributed        electronically or manufactured into printed items.    -   a. Parties and Events. Many events from weddings to birthday        parties comprise a series of photos that can be arranged into        common stories. Reunions are also great examples of events that        be arranged into common storybook templates.    -   b. Life Events. The various life events from the birth of a baby        to milestones are common and storybook templates can be helpful.    -   c. Yearbooks. The tradition of yearbooks for schools and        families can be distilled into common storybook templates that        include collections of photos.    -   d. Vacations. Many vacations from road trips to travel can be        compiled into common storybook templates.    -   e Seasonal/Holidays. Seasonal holiday traditions are shared by        many people and can be compiled into storybook templates.    -   f. Hobbies/Interests. Storybook templates could be designed        around many common interests such as sports and pets.    -   2. Features. One embodiment of the photo story book engine 1701        will include the following features. These features may be        implemented in computer software installed on a user's client        computer, online or embedded in cameras to guide the user to        capture photos based on a storybook template.    -   a. Integration with online photo storage services. As mentioned        above, one embodiment of the service 100 allows the user to        setup online photo accounts to automatically import the photos        from one or more external photo services 1706. In this        embodiment, the service 100 may detect when the user uploads new        photos (e.g., by periodically querying the external service        1706) and the photo story theme engine may responsively suggest        new photo storybooks that can be created with the new photos.        The photo story theme engine may also ask the user to add the        new photos to storybooks that the user is already working on.    -   b. Metadata and Time Series Analysis. In one embodiment, the        photo story theme engine 1701 will access the user's photos and        analyze them based on the available metadata and distribution of        photos taken over a period of time. This data is useful for        recommending storybook templates that will work best with the        user's photos. For example, some yearbook templates work best if        the user has some photos from each month of the year while other        templates work best if the user has most photos taken over a few        short intervals throughout the year (e.g., seasonal templates).    -   c. Template Variables. As mentioned, the storybook templates may        include variables that are replaced with information entered by        the user. There are several types of variables, which control        the input field format and how the variable data should be        processed. In one embodiment, the variable types are text, name        and date, but additional variable types are also contemplated.        The template variables may include format options to control how        the text is displayed. For example, format options for a name        variable may include first name, middle name, last name, first        initial, middle initial and last initial. Format options for        event date may include standard, month, day, year, and formal.    -   d. Template Conditionals. The storybook template can include        conditional logic to control the content based on input from the        user.    -   e. Collaborative Creation. In one embodiment, the user can allow        other users to add content to the storybook such as photos, text        and additional pages. The user can control what type of content        can be added by other users and where that content can be added.        For example, the user may allow each additional user to add one        page and the freedom to add text and photos to the page. As        another example, the user may allow other users to add only one        photo and text box to an existing page. The storybook template        can include suggestions for content from other users and preset        the permissions based on the story. For example, a party        storybook template might have a guests page where the user can        invite other users to add a note and photo.    -   f. Stock Photography. The service includes access to a database        of stock photos that can be used in storybooks. The stock photos        can complement the user's photos to help tell a story. For        example, a storybook about a Paris trip might include a stock        photo of the Eiffel Tower and the user can add their photos        taken there. The storybook template is setup to include a list        of stock photos based on tags or keywords.    -   g. Auto-Fill. After the user selects photos for a story or        entire storybook, the photo story theme engine may auto-fill the        photos into storybook pages based on the template layout, number        of photos, orientation of photos, and size of photos.    -   h. Screenshot Descriptions. An exemplary set of graphical user        interface features employed in one embodiment of the invention        are illustrated in FIGS. 19-28.

As illustrated in FIG. 19, the user may be presented with a list ofstorybook themes 1901 or occasions organized by category. For example,in one embodiment, the list includes the hierarchical categories listedabove. One embodiment also includes a search field (not shown) allowingthe user to search for a storybook template based on keywords.

After the user selects a storybook theme or occasion, he/she is providedwith a selectable list of one or more storybook templates 1902 for theselected theme. For example, for the Baby's First Year theme, theservice might have the templates listed above (i.e., “Classic Baby,” “ICan,” and “Baby Body Book”). After selecting a theme and template, anext button 1903 is provided to allow the user to proceed to the nextwindow.

As illustrated in FIG. 20, a storybook template can include textvariables 2001-2005 that are populated with values entered by the user.The illustrated GUI screenshot shows a form to enter the storybook name2001, the year 2002, a person 2003, honoree 2004 and event date 2005.

As mentioned, in one embodiment, a storybook template comprises one ormore stories and each story includes a name and questions orinstructions to guide the user in selecting the photos to fit the story.In the example shown in FIG. 21, the first story in the sequence is“Expecting Mommy” and the user is instructed to select pictures of theexpecting new mother. In one embodiment, each story is provided with itsown region 2101 within the GUI in which the user may add photos (e.g.,by clicking and dragging) and/or text comments. A second region 2102provides a graphical listing of relevant/filtered photos from the user'sphoto collection. In the example shown in FIG. 21, the photos have beenfiltered by a date range (i.e., Jan. 1, 2010 to Oct. 10, 2010) andsubject (the expecting mother). In one embodiment, the photos arefiltered using metadata associated with the photos such as date taken,and/or subject of the photos. As previously mentioned, the photos may bestored on the user's local computer system, on the online service 100,and/or on an external photo service 1706. The user may select photos forthe story by clicking on the thumbnail or dragging the photo to thestory area 2101. The user interface indicates which photos have beenpreviously selected for inclusion in the storybook (e.g., by displayingan image of the photo within the story region 2101).

FIG. 22 illustrates the GUI in which the user has entered the storybookname 2001, the honoree 2004, and the shower date 2005. This informationmay be used in the storybook according to the configuration of thestorybook template.

As illustrated in FIG. 23, after the user enters Kim Jones as thehonoree and Jan. 17, 2010 as the shower date, the service filters thephotos within region 2102 to those taken on Jan. 17, 2010. The user canalso filter photos to those tagged with Kim Jones. This screenshot alsoshows the second story in the template called “Decorations” withinregion 2101 and instructions for the user to include pictures of thedecorations. In addition, a text region 2301 is provided in which theuser may enter text associated with the second story.

FIG. 24 illustrates the third story in the template called “Treats”within region 2101 which includes instructions for the user to includepictures of the cake and other treats and another text region 2301 forentering text related to the “treats.” As with other stories in thestorybook template, the user can click on a thumbnail in the photo trayarea 2102 to add it to the story or drag the photo into the story area2101.

As illustrated in FIG. 25, the storybook template can be setup to allowthe user to add additional users using data entry region 2501 whichincludes a field for the guest name, email address and an “add guest”button. In one embodiment, the additional users are permitted to log inand add text and/or photos to the storybook. In this example, the“Guests” story asks the user to enter the name and email of each guestto include on the page. In one embodiment, the service 100 sends anemail to each guest to ask them to add a photo and message for thehonoree.

FIG. 26 illustrates another story in the storybook template with thetitle “Fun and Games” with instructions for the user to include photosof the games and other entertainment at the event.

FIG. 27 illustrates another story in the storybook template with thetitle “Group Shots” and instructions for the user to include pictures ofthe guests with the expecting mother.

In one embodiment, after the last story of the storybook template iscompleted by the user, the service displays a preview of the storybook.As illustrated in FIG. 28, the preview may be comprised of a simplelisting showing a separate entry 2801-2807 for each story.Alternatively, or in addition, the storybook may also be displayed in agraphical user interface that represents how the photos will appear inthe actual printed book.

System and Method for Generating Photo Stories Using Video

One embodiment of the invention comprises a system and method forgenerating story books using video. In one embodiment, the user ispermitted to upload a video meeting certain predefined criteria (e.g.,having specified size, format, and/or length constraints). Once the userhas uploaded a suitable video, the system automatically selects aplurality of image frames from the video to be used within the storybook. The number and separation of video image frames from the videostream may be selected based on user preferences and/or predefinedsystem parameters. For example, in one embodiment, if N images arerequested by the user for the story book, then N image frames, evenlyspaced in time across the entire video image may be selected.Alternatively, in one embodiment, the system analyzes the video streamto identify video images associated with different video “scenes.” Thismay be accomplished using known techniques in which changes betweensuccessive video frames are analyzed to identify the “scenes.”

Once an appropriate set of video images are identified and confirmed bythe user, one embodiment of the invention selects one or moreprospective story book templates. The user may specify a category to beused for selecting appropriate story book templates (e.g., Christmas,birthday, funny video sequence, etc.). The selected story book templatesare then displayed for review and/or editing by the end user. Forexample, in one embodiment, the user is provided with access to theselected image frames from the video stream along with the full set ofimage frames which were not selected. The user may then manually selectone or more unselected video images to be used in place of one or moreautomatically selected video images. In addition, in one embodiment, theuser is provided with the option to enter text, graphics or othercontent for each video image frame within the story book. The set ofcontent available to the user may be based on the particular story booktemplate. For example, a particular story book template may allow theuser to enter text or certain types of graphics for each selected videoimage frame within the story book. Once the story book template isprovided, video images and other content has been selected, the storybook is printed and mailed to a specified set of recipients.

As illustrated in FIG. 29, one embodiment of a system for creating photostories using video comprises video processing module 2901 forprocessing video streams uploaded by users. For example, in oneembodiment, the video processing module 2901 extracts a set of videoimage frames from the video stream and stores the extracted images 2902within the service database 115. As described above, in one embodiment,the video processing module 2901 is configured to select a set of Nevenly temporally spaced video image frames, where N is a number whichis either specified by the end user and/or specified by the video photostory layout engine 2903. For example, if the user first selects aparticular story book template which requires a specified number ofvideo image frames (or a specified range of video image frames), then Nwill be selected based on the specified number of video image frames. Inone embodiment, rather than merely selecting N evenly spaced frames, thevideo processing module 2901 includes scene detection logic 2910 fordetecting and extracting video image frames associated with differentscenes within the video stream.

Rather than initially selecting a story book template, the user mayrequest that the video photo story layout engine 2903 provide a previewof various different story book templates, each utilizing a differentnumber of video frames (i.e., each having a different value of N). Insuch a case, different values of N may be automatically selected for thedifferent templates. The different story book templates may then bepreviewed by the user with the N different video frames.

Once an appropriate set of video images are identified, the video photostory layout engine 2903 may provide the user with different templateoptions. The user will then be allowed to edit the template by addingtext, graphics or other objects. Once the template is complete, thecompleted photo 2904 is stored within the service database 115 and aprint job 255 is generated (as described in the related applications).The user may then identify the recipients to receive copies of theprinted photo story.

FIG. 30 illustrates a method for creating a photo story using video inaccordance with one embodiment of the invention. The method may beimplemented within the context of the architecture shown in FIG. 29, butis not limited to such an architecture.

At 3000, a user request to build a story book using a video stream isreceived. At 3001, the video stream is analyzed to determine if requiredspecifications are met. For example, limits may be placed on the size,format, and/or duration or the video clip. In one embodiment, variousdifferent video formats are supported such as MPEG-4 and audio videointerleave (AVI). However, the underlying principles of the inventionare not limited to any particular video format. If the video streamspecifications are not met, determined at 3002, then at 2010 the user isinformed and the user is given an opportunity to modify the video andtry again. For example, the user may decrease the size of the videostream (e.g., by decreasing the resolution or length of the videostream). In one embodiment, client software such as a browser plugin mayreformat the video stream in response to a user request to do so (e.g.,clipping the video stream to a predetermined maximum length or size orconverting the bitrate or format of the video stream).

Assuming that the video stream specifications are met then, at 3003, thevideo stream is uploaded. At 3004, the video stream is analyzed toidentify video images for extraction. In one embodiment, the user mayspecify that video images are to be automatically identified andextracted (e.g., by choosing a video frame periodically or by usingscene detection logic as described above). Alternatively, the user maymanually select the video frames. At 3005, the specified video framesare extracted from the video stream and, at 3006, the extracted videostreams are provided to the user for review and/or editing. For example,as mentioned, the selected video frames may be provided along with oneor more unselected video frames. The user may then choose one or more ofthe unselected video frames in place of one or more of the selectedframes.

In one embodiment, if the video stream is coded with MPEG-2, MPEG-4 orany other inter-frame and intra-frame coding, then only intra-framecoding frames (such as I-frames) may be selected to be used in the storybook. However, the underlying principles of the invention are notlimited to any particular frame type or frame coding technique.

At 3007, one or more story book templates are selected and the videoimages are added to the templates. A preview of the story book templatesis then provided to the end user for review and/or editing. For example,as previously mentioned, the user may be provided with the ability toadd text or move the positioning of the video frames within the storybook template. Once the user has confirmed that the story book iscomplete, at 3008 a print job for the story book is generated. Theuser's address book may be used to identify recipients for the storybook or the user may manually enter addresses of the recipients.Finally, at 3009, the story book is printed and mailed to the selectedrecipients.

As illustrated in FIG. 31, one embodiment of a graphical user interface(GUI) for generating a story book template comprises a set of story bookpages 3110 containing video frames 3101, 3103, 3106, 3109. The videoframes may initially be selected by the video photo story layout engine3903 using the techniques described above. Story book pages 3 through 5are illustrated in FIG. 31 but the user may cause other story book pagesto be displayed by selecting left arrow 3120 or the right arrow 3121(e.g., the left arrow to display story book pages 1 and 2 and the rightarrow to display story book pages 6 and higher).

A complete sequence of video frames 3111 is displayed beneath the storybook pages. In one embodiment, the “complete” set includes onlyinfra-frame coded frames (e.g., I-frames) of the video stream. The usermay scroll to earlier frames in the sequence using a left arrow 3122 andmay scroll to later frames in the sequence using the right arrow 3123.As indicated by the dotted arrows, the user may click and drag a framefrom the video stream 3111 to the story book pages to replace a currentvideo frame displayed within the story book pages with the selectedvideo frame (i.e., to replace a frame automatically selected by thesystem or previously selected by the user). In the example, the user hasmoved video frames 3103 and 3106 to story book pages 3 and 4,respectively. In one embodiment, each story book page has pre-allocatedlocations for each video frame. Alternatively, or in addition, certainstory book pages may allow the user to place video frames in any desiredposition. One embodiment also provides the user with editing tools(e.g., implemented as a browser applet/plugin) to resize, crop, providespecial affects and/or adjust the color of each frame. As illustrated,text regions 3125-3126 may also be provided to allow the user to entertext on certain story book pages. In one embodiment, the user may clickon a text region to enter/edit the text. Once the user has completedediting the story book preview, he/she may click on a “complete” button3130 to begin the printing/distribution process.

In one embodiment, the different graphical user interface (GUI) featuresdescribed herein are generated by presentation and session managementlogic 106 executed on the online stationery service. In one embodiment,various well known functional modules associated within the presentationand session management logic 106 are executed to receive input, processthe input, interact with one or more other modules shown in the figures,and dynamically generate Web pages containing the results. The Web pagesare then transmitted to the user's client computer 140 and rendered on abrowser 145. The Web pages may be formatted according to the HyperTextMarkup Language (“HTML”) or Extensible HTML (“XHTML”) formats, and mayprovide navigation to other Web pages via hypertext links. Oneembodiment utilizes Dynamic HTML (“DHTML”), a collection of technologiesused together to create interactive Web sites by using a combination ofa static markup language (e.g., HTML), a client-side scripting language(e.g., JavaScript), a presentation definition language (e.g., CSS), andthe Document Object Model (“DOM”). Note that in some figures andassociated description (e.g., FIG. 29), the presentation and sessionmanagement logic is not illustrated or described to avoid obscuring theunderlying principles of the invention.

Throughout the discussion above, various details have been omitted toavoid obscuring the pertinent aspects of the invention. For example, inan embodiment of the invention in which the user connects to the onlinephoto service 100 via a Web browser, various well known functionalmodules associated within the presentation and session management logic206 shown in the figures are executed to receive input, process theinput and dynamically generate Web pages containing the results. The Webpages described herein may be formatted according to the well knownHyperText Markup Language (“HTML”) or Extensible HTML (“XHTML”) formats,and may provide navigation to other Web pages via hypertext links. Oneembodiment utilizes Dynamic HTML (“DHTML”), a collection of technologiesused together to create interactive Web sites by using a combination ofa static markup language (e.g., HTML), a client-side scripting language(e.g., JavaScript), a presentation definition language (e.g., CSS), andthe Document Object Model (“DOM”). Of course, the underlying principlesof the invention are not limited to any particular set of protocols orstandards.

In one embodiment, the Web server used to implement the embodiments ofthe invention is an Apache web server running on Linux with softwareprogrammed in PHP using a MySQL database.

Embodiments of the invention may include various steps as set forthabove. The steps may be embodied in machine-executable instructionswhich cause a general-purpose or special-purpose processor to performcertain steps. Alternatively, these steps may be performed by specifichardware components that contain hardwired logic for performing thesteps, or by any combination of programmed computer components andcustom hardware components.

Elements of the present invention may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, floppydiskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs,RAMs, EPROMs, EEPROMs, magnetic or optical cards, propagation media orother type of media/machine-readable medium suitable for storingelectronic instructions. For example, the present invention may bedownloaded as a computer program which may be transferred from a remotecomputer (e.g., a server) to a requesting computer (e.g., a client) byway of data signals embodied in a carrier wave or other propagationmedium via a communication link (e.g., a modem or network connection).

Throughout the foregoing description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art that the invention may be practiced without some ofthese specific details. For example, it will be readily apparent tothose of skill in the art that the functional modules such as wizardsand other logic may be implemented as software, hardware or anycombination thereof. Accordingly, the scope and spirit of the inventionshould be judged in terms of the claims which follow.

What is claimed is:
 1. A computer-implemented method for creating aphoto book using a video stream comprising: determining, by a computersystem, whether a video stream meets one or more predeterminedrequirements for producing a photo book, wherein the video streamcomprises a plurality of video frames, wherein the video streamcomprises intra-frame image coding which comprises I-frame; analyzingthe video stream, by the computer system, to select some of theplurality of video frames as selected video frames based on a set ofpredetermined frame selection criteria; extracting the selected videoframes from the video stream; providing a story book template by thecomputer system; generating a preview of the story book template usingthe selected video frames within a graphical user interface (GUI),wherein the story book template comprises one or more book pages eachconfigured to incorporate one or more of the selected video frames; andproducing a design of photo book based on the story book template thatincorporates the selected video frames.
 2. The computer-implementedmethod as in claim 1, wherein the one or more predetermined requirementscomprise size, format, or duration of a video stream.
 3. Thecomputer-implemented method as in claim 1, wherein the computer systemselects evenly spaced video frames in the video stream in the step ofanalyzing the video stream.
 4. The computer-implemented method as inclaim 1, further comprising: detecting scenes in the plurality of videoframes; and selecting some of the plurality of video frames based on thescenes detected in the plurality of video frames.
 5. Thecomputer-implemented method as in claim 1, wherein the step of providinga story book template comprises: suggesting to a user, by the computersystem, one or more story book templates based on the video frame. 6.The computer-implemented method as in claim 1, wherein the plurality ofvideo frames from the video stream are displayed on the GUI, wherein thevideo frames are selectable and configured to be incorporated in thebook pages.
 7. The computer-implemented method as in claim 6, furthercomprising: allowing a user to incorporate one of the plurality of videoframes to one of the one or more book pages.
 8. The computer-implementedmethod as in claim 1, further comprising: allowing a user to edit thestory book template.
 9. The computer-implemented method as in claim 8,further comprising: allowing the user to add text to the story booktemplate.
 10. The computer-implemented method as in claim 1, furthercomprising: producing a hardcopy photo book based on the design of photobook.
 11. A computer-implemented method for creating a photo book usinga video stream comprising: determining, by a computer system, whether avideo stream meets one or more predetermined requirements for producinga photo book, wherein the video stream comprises a plurality of videoframes, wherein the video stream comprises inter-frame coding; analyzingthe video stream, by the computer system, to select some of theplurality of video frames as selected video frames based on a set ofpredetermined frame selection criteria; extracting the selected videoframes from the video stream; providing a story book template by thecomputer system; generating a preview of the story book template usingthe selected video frames within a graphical user interface (GUI),wherein the story book template comprises one or more book pages eachconfigured to incorporate one or more of the selected video frames; andproducing a design of photo book based on the story book template thatincorporates the selected video frames.
 12. The computer-implementedmethod as in claim 11, wherein the one or more predeterminedrequirements comprise size, format, or duration of a video stream. 13.The computer-implemented method as in claim 11, wherein the computersystem selects evenly spaced video frames in the video stream in thestep of analyzing the video stream.
 14. The computer-implemented methodas in claim 11, further comprising: detecting scenes in the plurality ofvideo frames; and selecting some of the plurality of video frames basedon the scenes detected in the plurality of video frames.
 15. Thecomputer-implemented method as in claim 11, wherein the step ofproviding a story book template comprises: suggesting, by the computersystem to a user, one or more story book templates based on the videoframe.
 16. The computer-implemented method as in claim 11, wherein theplurality of video frames from the video stream are displayed on theGUI, wherein the video frames are selectable and configured to beincorporated in the book pages.
 17. The computer-implemented method asin claim 16, further comprising: allowing a user to incorporate one ofthe plurality of video frames to one of the one or more book pages. 18.The computer-implemented method as in claim 11, further comprising:allowing a user to edit the story book template.
 19. Thecomputer-implemented method as in claim 18, further comprising: allowingthe user to add text to the story book template.
 20. Thecomputer-implemented method as in claim 11, further comprising:producing a hardcopy photo book based on the design of photo book.